{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Warm Start from a Completed Hyper-Parameter Tuning Job\n",
    "Once the previous hyper-parameter tuning job completes, we can perform another round of optimization using `Warm Start`."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Warm start configuration allows you to create a new tuning job with the learning gathered in a parent tuning job by specifying up to 5 parent tuning jobs. If a warm start configuration is specified, Automatic Model Tuning will load the previous [hyperparameter set, objective metrics values] to warm start the new tuning job. This means, you can continue optimizing your model from the point you finished your previous tuning job experiment."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "<img src=\"img/hpt-warmstart.png\" width=\"90%\" align=\"left\">"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import boto3\n",
    "import sagemaker\n",
    "import pandas as pd\n",
    "\n",
    "sess = sagemaker.Session()\n",
    "bucket = sess.default_bucket()\n",
    "role = sagemaker.get_execution_role()\n",
    "region = boto3.Session().region_name\n",
    "\n",
    "sm = boto3.Session().client(service_name=\"sagemaker\", region_name=region)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pre-Requisite\n",
    "\n",
    "## The previous hyper-parameter tuning job needs to complete, before we can perform another round of optimization using `Warm Start`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r tuning_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    tuning_job_name\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the previous Hyperparameter Tuning notebook.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(tuning_job_name)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Check the status of the previous Hyperparameter Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "job_description = sm.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuning_job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "if not bool(job_description):\n",
    "    print(\"+++++++++++++++++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the previous Hyperparameter Tuning notebook before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++++++++++++++++\")\n",
    "elif job_description[\"HyperParameterTuningJobStatus\"] == \"Completed\":\n",
    "    print(\"[OK] Previous Tuning Job has completed. Please continue.\")\n",
    "else:\n",
    "    print(\"+++++++++++++++++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the previous Hyperparameter Tuning notebook.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Specify the S3 Location of the Features"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r processed_train_data_s3_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    processed_train_data_s3_uri\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the notebooks in the previous PREPARE section before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(processed_train_data_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r processed_validation_data_s3_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    processed_validation_data_s3_uri\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the notebooks in the previous PREPARE section before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(processed_validation_data_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r processed_test_data_s3_uri"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    processed_test_data_s3_uri\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the notebooks in the previous PREPARE section before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(processed_test_data_s3_uri)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(processed_train_data_s3_uri)\n",
    "!aws s3 ls $processed_train_data_s3_uri/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(processed_validation_data_s3_uri)\n",
    "!aws s3 ls $processed_validation_data_s3_uri/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(processed_test_data_s3_uri)\n",
    "!aws s3 ls $processed_test_data_s3_uri/"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.inputs import TrainingInput\n",
    "\n",
    "s3_input_train_data = TrainingInput(s3_data=processed_train_data_s3_uri, distribution=\"ShardedByS3Key\")\n",
    "s3_input_validation_data = TrainingInput(s3_data=processed_validation_data_s3_uri, distribution=\"ShardedByS3Key\")\n",
    "s3_input_test_data = TrainingInput(s3_data=processed_test_data_s3_uri, distribution=\"ShardedByS3Key\")\n",
    "\n",
    "print(s3_input_train_data.config)\n",
    "print(s3_input_validation_data.config)\n",
    "print(s3_input_test_data.config)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "!cat src/tf_bert_reviews.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Hyper-Parameters for Classification Layer\n",
    "First, retrieve `max_seq_length` from the prepare phase."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r max_seq_length"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    max_seq_length\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the notebooks in the previous PREPARE section before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(max_seq_length)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "epochs = 3\n",
    "epsilon = 0.00000001\n",
    "train_batch_size = 128\n",
    "validation_batch_size = 128\n",
    "test_batch_size = 128\n",
    "train_steps_per_epoch = 100\n",
    "validation_steps = 100\n",
    "test_steps = 100\n",
    "train_instance_count = 1\n",
    "train_instance_type = \"ml.c5.4xlarge\"\n",
    "train_volume_size = 1024\n",
    "use_xla = True\n",
    "use_amp = True\n",
    "enable_sagemaker_debugger = False\n",
    "enable_checkpointing = False\n",
    "enable_tensorboard = False\n",
    "input_mode = \"File\"\n",
    "run_validation = True\n",
    "run_test = True\n",
    "run_sample_predictions = True"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Track the Optimizations Within our Experiment"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r experiment_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    experiment_name\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the notebooks in the previous TRAIN section before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(experiment_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store -r trial_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "try:\n",
    "    trial_name\n",
    "    print(\"[OK]\")\n",
    "except NameError:\n",
    "    print(\"+++++++++++++++++++++++++++++++\")\n",
    "    print(\"[ERROR] Please run the notebooks in the previous TRAIN section before you continue.\")\n",
    "    print(\"+++++++++++++++++++++++++++++++\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(trial_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "import time\n",
    "from smexperiments.trial import Trial\n",
    "\n",
    "timestamp = \"{}\".format(int(time.time()))\n",
    "\n",
    "trial = Trial.load(trial_name=trial_name)\n",
    "print(trial)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from smexperiments.tracker import Tracker\n",
    "\n",
    "tracker_optimize = Tracker.create(display_name=\"optimize-2\", sagemaker_boto_client=sm)\n",
    "\n",
    "optimize_trial_component_name = tracker_optimize.trial_component.trial_component_name\n",
    "print(\"Optimize trial component name {}\".format(optimize_trial_component_name))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Attach the `deploy` Trial Component and Tracker as a Component to the Trial"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "trial.add_trial_component(tracker_optimize.trial_component)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Dynamic Hyper-Parameter Ranges to Explore\n",
    "While not necessary, we can choose to statically define any hyper-parameters that we are not choosing to explore in this WarmStart optimization run.\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import IntegerParameter\n",
    "from sagemaker.tuner import ContinuousParameter\n",
    "from sagemaker.tuner import CategoricalParameter\n",
    "from sagemaker.tuner import HyperparameterTuner\n",
    "\n",
    "hyperparameter_ranges = {\n",
    "    \"learning_rate\": ContinuousParameter(0.00015, 0.00075, scaling_type=\"Linear\"),\n",
    "    \"train_batch_size\": CategoricalParameter([64, 128]),\n",
    "    \"freeze_bert_layer\": CategoricalParameter([True, False]),\n",
    "}"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Track the Hyper-Parameter Ranges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker_optimize.log_parameters(hyperparameter_ranges)\n",
    "\n",
    "# must save after logging\n",
    "tracker_optimize.trial_component.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Metrics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "metrics_definitions = [\n",
    "    {\"Name\": \"train:loss\", \"Regex\": \"loss: ([0-9\\\\.]+)\"},\n",
    "    {\"Name\": \"train:accuracy\", \"Regex\": \"accuracy: ([0-9\\\\.]+)\"},\n",
    "    {\"Name\": \"validation:loss\", \"Regex\": \"val_loss: ([0-9\\\\.]+)\"},\n",
    "    {\"Name\": \"validation:accuracy\", \"Regex\": \"val_accuracy: ([0-9\\\\.]+)\"},\n",
    "]"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tensorflow import TensorFlow\n",
    "\n",
    "estimator = TensorFlow(\n",
    "    entry_point=\"tf_bert_reviews.py\",\n",
    "    source_dir=\"src\",\n",
    "    role=role,\n",
    "    instance_count=train_instance_count,  # Make sure you have at least this number of input files or the ShardedByS3Key distibution strategy will fail the job due to no data available\n",
    "    instance_type=train_instance_type,\n",
    "    volume_size=train_volume_size,\n",
    "    py_version=\"py37\",\n",
    "    framework_version=\"2.3.1\",\n",
    "    hyperparameters={\n",
    "        \"epochs\": epochs,\n",
    "        \"epsilon\": epsilon,\n",
    "        \"validation_batch_size\": validation_batch_size,\n",
    "        \"test_batch_size\": test_batch_size,\n",
    "        \"train_steps_per_epoch\": train_steps_per_epoch,\n",
    "        \"validation_steps\": validation_steps,\n",
    "        \"test_steps\": test_steps,\n",
    "        \"use_xla\": use_xla,\n",
    "        \"use_amp\": use_amp,\n",
    "        \"max_seq_length\": max_seq_length,\n",
    "        \"enable_sagemaker_debugger\": enable_sagemaker_debugger,\n",
    "        \"enable_checkpointing\": enable_checkpointing,\n",
    "        \"enable_tensorboard\": enable_tensorboard,\n",
    "        \"run_validation\": run_validation,\n",
    "        \"run_test\": run_test,\n",
    "        \"run_sample_predictions\": run_sample_predictions,\n",
    "    },\n",
    "    input_mode=input_mode,\n",
    "    metric_definitions=metrics_definitions,\n",
    "    #                       max_run=7200 # max 2 hours * 60 minutes seconds per hour * 60 seconds per minute\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup Warm Start Config\n",
    "We configure `WarmStartConfig` using 1 or more  of the previous hyper-parameter tuning job runs called the `parent` jobs - as well as a `WarmStartType`.  The parents must have finished either with one of the following success or failure states: `Completed`, `Stopped`, or `Failed`.\n",
    "\n",
    "`WarmStartType` is one of the following strategies:\n",
    "\n",
    "* `IDENTICAL_DATA_AND_ALGORITHM` uses the same input data and algorithm as the parent tuning jobs, but allows a practitioner to explore more hyper-parameter range values.  Upon completion, a tuning job with this strategy will return an additional field, `OverallBestTrainingJob` containing the best model candidate including this tuning job as well as the completed parent tuning jobs.\n",
    "* `TRANSFER_LEARNING` allows you to transfer the knowledge from previous tuning jobs.  You can use different input dataset and algorithm - as well as everything from the `IDENTICAL_DATA_AND_ALGORITHM` strategy.\n",
    "\n",
    "_Note:  Recursive parent-child relationships are not supported._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(\"Previous Tuning Job Name: {}\".format(tuning_job_name))"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.tuner import WarmStartConfig\n",
    "from sagemaker.tuner import WarmStartTypes\n",
    "\n",
    "warm_start_config = WarmStartConfig(\n",
    "    warm_start_type=WarmStartTypes.IDENTICAL_DATA_AND_ALGORITHM, parents={tuning_job_name}\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Setup HyperparameterTuner with Warm Start Config including New Hyper-Parameter Ranges"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "objective_metric_name = \"train:accuracy\"\n",
    "\n",
    "tuner = HyperparameterTuner(\n",
    "    estimator=estimator,\n",
    "    objective_type=\"Maximize\",\n",
    "    objective_metric_name=objective_metric_name,\n",
    "    hyperparameter_ranges=hyperparameter_ranges,\n",
    "    metric_definitions=metrics_definitions,\n",
    "    max_jobs=2,\n",
    "    max_parallel_jobs=1,\n",
    "    strategy=\"Bayesian\",\n",
    "    early_stopping_type=\"Auto\",\n",
    "    warm_start_config=warm_start_config,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Start Tuning Job"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "tuner.fit(\n",
    "    {\"train\": s3_input_train_data, \"validation\": s3_input_validation_data, \"test\": s3_input_test_data},\n",
    "    include_cls_metadata=False,\n",
    "    wait=False,\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# If You See an Error, Please Wait for the Hyper-Parameter Tuning Job to Complete from the Previous Notebook"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "##  Check Tuning Job Status\n",
    "\n",
    "Re-run this cell to track the status."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from pprint import pprint\n",
    "\n",
    "tuning_job_name = tuner.latest_tuning_job.job_name\n",
    "\n",
    "job_description = sm.describe_hyper_parameter_tuning_job(HyperParameterTuningJobName=tuning_job_name)\n",
    "\n",
    "status = job_description[\"HyperParameterTuningJobStatus\"]\n",
    "\n",
    "print(\"\\n\")\n",
    "print(status)\n",
    "print(\"\\n\")\n",
    "pprint(job_description)\n",
    "\n",
    "if status != \"Completed\":\n",
    "    job_count = job_description[\"TrainingJobStatusCounters\"][\"Completed\"]\n",
    "    print(\"Not yet complete, but {} jobs have completed.\".format(job_count))\n",
    "\n",
    "    if job_description.get(\"BestTrainingJob\", None):\n",
    "        print(\"Best candidate:\")\n",
    "        pprint(job_description[\"BestTrainingJob\"][\"TrainingJobName\"])\n",
    "        pprint(job_description[\"BestTrainingJob\"][\"FinalHyperParameterTuningJobObjectiveMetric\"])\n",
    "    else:\n",
    "        print(\"No training jobs have reported results yet.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "from IPython.core.display import display, HTML\n",
    "\n",
    "display(\n",
    "    HTML(\n",
    "        '<b>Review <a target=\"blank\" href=\"https://console.aws.amazon.com/sagemaker/home?region={}#/hyper-tuning-jobs/{}\">Hyper-Parameter Tuning Job</a></b>'.format(\n",
    "            region, tuning_job_name\n",
    "        )\n",
    "    )\n",
    ")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# _Please Wait for the ^^ Tuning Job ^^ to Complete Above_"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tuner.wait()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show the Tuning Job\n",
    "### _Note:  This will fail at first.  Please wait about 15-30 seconds and re-run._"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import HyperparameterTuningJobAnalytics\n",
    "\n",
    "hp_results = HyperparameterTuningJobAnalytics(sagemaker_session=sess, hyperparameter_tuning_job_name=tuning_job_name)\n",
    "\n",
    "df_results = hp_results.dataframe()\n",
    "df_results.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "df_results.sort_values(\"FinalObjectiveValue\", ascending=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show the Overall Best Candidate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "df_results.sort_values(\"FinalObjectiveValue\", ascending=0).head(1)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "best_candidate_tuning_job_name = df_results.sort_values(\"FinalObjectiveValue\", ascending=0).head(1)[\"TrainingJobName\"]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Log the Best Hyper-Parameter and Objective Metric in the Experiment\n",
    "\n",
    "Logging `learning_rate` parameter and `accuracy` metric"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_learning_rate = df_results.sort_values(\"FinalObjectiveValue\", ascending=0).head(1)[\"learning_rate\"]\n",
    "print(best_learning_rate)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "best_accuracy = df_results.sort_values(\"FinalObjectiveValue\", ascending=0).head(1)[\"FinalObjectiveValue\"]\n",
    "print(best_accuracy)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker_optimize.log_parameters({\"learning_rate\": float(best_learning_rate)})\n",
    "\n",
    "# must save after logging\n",
    "tracker_optimize.trial_component.save()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tracker_optimize.log_metric(\"accuracy\", float(best_accuracy))\n",
    "\n",
    "tracker_optimize.trial_component.save()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## _Ignore any ^^ ERROR above ^^. This is OK._"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Show Experiment Analytics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sagemaker.analytics import ExperimentAnalytics\n",
    "\n",
    "lineage_table = ExperimentAnalytics(\n",
    "    sagemaker_session=sess,\n",
    "    experiment_name=experiment_name,\n",
    "    metric_names=[\"validation:accuracy\"],\n",
    "    sort_by=\"CreationTime\",\n",
    "    sort_order=\"Descending\",\n",
    ")\n",
    "\n",
    "lineage_df = lineage_table.dataframe()\n",
    "lineage_df.shape"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "lineage_df"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Pass `tuning_job_name` to the Next Notebook"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "print(best_candidate_tuning_job_name)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%store best_candidate_tuning_job_name"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "%store"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Release Resources"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%html\n",
    "\n",
    "<p><b>Shutting down your kernel for this notebook to release resources.</b></p>\n",
    "<button class=\"sm-command-button\" data-commandlinker-command=\"kernelmenu:shutdown\" style=\"display:none;\">Shutdown Kernel</button>\n",
    "        \n",
    "<script>\n",
    "try {\n",
    "    els = document.getElementsByClassName(\"sm-command-button\");\n",
    "    els[0].click();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}    \n",
    "</script>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%javascript\n",
    "\n",
    "try {\n",
    "    Jupyter.notebook.save_checkpoint();\n",
    "    Jupyter.notebook.session.delete();\n",
    "}\n",
    "catch(err) {\n",
    "    // NoOp\n",
    "}"
   ]
  }
 ],
 "metadata": {
  "instance_type": "ml.t3.medium",
  "kernelspec": {
   "display_name": "Python 3 (Data Science)",
   "language": "python",
   "name": "python3__SAGEMAKER_INTERNAL__arn:aws:sagemaker:us-east-1:081325390199:image/datascience-1.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}
